LLM Document Summarizer — LLMOps with Amazon Bedrock
A production document summarization system demonstrating four LLMOps patterns: prompt versioning, automated evaluation, guardrails, and experiment tracking.
Prompt templates are stored as versioned YAML files (v1, v2, v3) in a prompts/ directory. Switching the active version is a single config change — no code deployment required. An evaluation pipeline (evaluate.py) runs all prompt versions against a held-out set of documents with human-written reference summaries and computes ROUGE scores. Results are logged to MLflow alongside every production inference call — latency, prompt version, guardrails activity, and input/output length.
Bedrock Guardrails are applied to every response: PII (names, emails, phone numbers) is anonymised before the summary reaches the user, and harmful content is blocked. Every guardrail action is logged to MLflow for monitoring.
Served via FastAPI with auto-generated OpenAPI docs at /docs. Runs locally via Docker Compose (API + MLflow tracking server). GitHub Actions CI runs pytest on every push.
Quick Facts
Overview
Problem
Demonstrating the gap between an LLM wrapper (call an API, return the result) and a production LLMOps system (version prompts, evaluate quality, enforce guardrails, track everything) — and implementing all four production patterns in a single deployable application.
Solution
Three versioned prompt templates in prompts/. evaluate.py runs ROUGE scoring across all versions on a committed eval set and logs to MLflow. bedrock_client.py applies Bedrock Guardrails to every call and logs all metadata. FastAPI serves POST /summarize (PDF upload) and POST /summarize/text. Docker Compose runs API + MLflow server.
Challenges
Bedrock Guardrails require a guardrail to be created in the AWS console before the project runs — the guardrail ID is a config value, not a code value. Input content filtering blocked legitimate documents (CVs with disability certificates, phone numbers, personal data) even at MEDIUM sensitivity — the fix was to disable input content filtering entirely and filter only the output. Bedrock does not return a guardrail action header on invoke_model responses; blocked inputs return plain text instead of JSON, requiring the body text to be checked before JSON parsing. ROUGE metrics measure lexical overlap, not semantic quality — the eval set and reference summaries must be written carefully to make the scores meaningful. Prompt templates use Python .format()-style placeholders inside JSON structures, which requires escaping curly braces in the JSON schema to prevent format errors.
Results / Metrics
Evaluated three prompt versions across five held-out documents (business report, research abstract, news article, legal notice, technical README).
ROUGE-L scores: v1=0.497, v2=0.379, v3=0.504 — ROUGE-1: v1=0.595, v2=0.532, v3=0.609 — ROUGE-2: v1=0.352, v2=0.249, v3=0.350. Best prompt: v3 (ROUGE-L 0.504) — extended prompt with risk flags and audience targeting consistently outperformed v1 and v2 across document types. Average latency: v1=3857ms, v2=4170ms, v3=5105ms. Guardrails firing rate: 0/15 calls on the eval set. Active prompt updated to v3.
Screenshots
Click to enlarge.
Click to enlarge.
